Here's the lowdown on two tools that track and measure the entire life of a development project.

Two weeks ago, I was watching TV, and half of the living room lights kept turning off 16 minutes after I turned them on. This was the third time I'd been through this, and this time I wrote a sticky note to remind myself to fix the problem. I then affixed the note to the banister leading upstairs and when I went to bed, I dutifully picked up the note, carried it upstairs, and transferred it to my computer monitor. The next evening, I fixed the problem.

I've recently gotten involved in home automation, and the lights were turning off because of a bug in a script I'd written. But I watch TV only on the weekends. And I turn on all the lights in the living room even less often—as I recall, only on Sundays since getting the automation gear. So on Monday morning, I'd go to work and forget about the problem. What I needed was a system for reporting bugs to myself, and, in this case, a PostIt note proved to be a usable solution.

This wasn't the first time I'd used a sticky note to report a bug to myself, and I've found them to be an effective communications medium—from me to me. But something more sophisticated is required when the development team is larger than one.

The Basics and Beyond
Even a team doesn't have to use a dedicated program to track bugs and features. You can often do an effective job using a simple database or even a spreadsheet—provided the team and projects are small enough. But once a development project grows beyond two or three developers and starts to involve dedicated QA and tech support departments, simplistic solutions quickly collapse under the weight of the added complexity, and it's time to look for a proper tracking program. The question then becomes what to look for.

Tracking bugs and identifying their priority and status is an obvious necessity. You'll also want a description of the bug and its fix, the module it occurred in, the name of the developer who reported it (in case questions arise) and so on. If you're introducing a new tool or process into your development cycle, it may seem best to begin simply, using just the basics, but before long you'll start thinking of other things you'd like to do, so why not consider them up front? And if you're already using such a tool, you may want more capabilities or flexibility.

For example, a powerful reporting capability will give you feedback on the development process itself. Trend reports are particularly valuable for such insights. As the number of bugs reported declines during development, you'll get a good indication of the time you can expect to ship alpha and then beta software for user testing. In turn, that feedback will help to predict the final release date. The ability to see the timeline unfolding is far more useful than simple totals.

You'll also want to prioritize bugs in various degrees, from "show-stopper" to "fix it if you get a chance," and to categorize bugs in various ways. I've found it useful to be able to specify the area of the program a bug appears in. In one particular case, an inordinate number of bugs (compared with other areas of the program) were appearing in the memory management modules. We had a team meeting about the problem and decided the culprit wasn't the software itself, but the fact that the hardware design made the software tremendously complex. A few tweaks in the circuitry, a new implementation of the memory management software, and we were back in business.

Customization is also a key feature. You should be able to specify the number and names of items like priorities, categories, status and so on—a bug tracker shouldn't dictate these important issues. Also, I like being able to keep track of the phase when a bug appears in the program. This doesn't mean when it was reported; rather, whether the bug is caused by incorrect requirements, bad design, poor coding and so on. This information is tremendously helpful in the post-mortem stage of a project when you want to learn from your mistakes. And if you're adding fields to the database, you had better be able to customize existing reports and even create new ones.

As long as you're trying to gain insight into the development process, you might as well not limit yourself to bugs: Go all the way and start tracking code from the very beginning of the project. So, what I'm really describing here is a life-cycle development database. And, given that, it would be great if the developers could integrate it with their version control system (VCS), and if QA could integrate it with their testing software.

Next, you need a means of assigning features and bugs for coding, repair and testing. The capacity to make assignments requires some sort of hierarchy, and that depends on the ability to determine who can do what when. Again, this database should be as configurable as possible, so you can adjust it to meet your team's needs, organization and existing processes.

Lastly, this is, after all, the Internet age, so it would be nice to have Internet support. Ideally, all desktop functions should be available through a browser. Development teams are growing more and more distributed, and although virtual private networks (VPNs) are becoming more prevalent, you might have contract workers, customers and others who need access to the database, but whom you don't want to have internal network access.

DevTrack
I was pleased to find a printed manual when I opened the package from TechExcel containing DevTrack 4.1. Its organization is a bit strange, with addendums at the front of the manual specifying what's changed since version 3.0, but still, an actual paper manual—cool! Well, not so cool after all. The tutorial section is either out-of-date or was filled with errors to begin with. For instance, the name you're told to use to log into the sample database isn't valid, and a folder supposedly named QA-Ready for Testing didn't exist, either. Despite a recommendation to update the "Current Status Page frequently," I couldn't find anything named Current Status Page. The result was that I had to have a decent grasp of the program before I could make the tutorial work.

Although the DevTrack manual and sample database focus on bug tracking, the program is certainly capable of working as a complete life-cycle tracker, and the documentation refers to OWL (which stands for Open Workflow and Life cycle). The idea is that you define your development workflow process and then configure DevTrack to support it. (A white paper available at www.techexcel.com/company/whitepaper/DevTrackWP.html provides a detailed explanation.)

For example, at the beginning of a project, the project manager can enter a set of specific features to be implemented. These can be assigned to developers immediately or can reside in a common Developers folder—either to be assigned later, or for programmers to assign to themselves or each other, depending on how you configure permissions. As each feature is implemented, the developer forwards it to a QA folder, where testing personnel can access the item—again, assignment is configurable. If the feature has a bug, the tester can generate a bug issue, link it to the original feature and route it back to Developers to be fixed. The ability to assign and forward issues to individuals and folders goes a long way toward organizing a workflow.

Even better is DevTrack's ability to define as many folders as you need, specifying access permissions according to the particular group to which an individual belongs. The groups are also definable. You can also specify "applicable owners" for issues depending on their current status; so, perhaps, a member of the developer group is not an applicable owner for an item that is Ready for Testing. Besides the applicable owner, the program enables you to specify which state transitions are legal. For example, you might specify that an item can go from Ready for Coding to Coding, but not directly to Ready for Testing. With these two capabilities you really do have the ability to develop a true workflow.

The program also supports an Auto-Route feature that can automatically assign issues to individuals or folders based on current workload, skill level or a weighted combination of the two. I'm not sure how useful this particular capability is because the factors determining who does what when tend to be more complicated and dynamic than simple skill level or workload allow for.

DevTrack offers a wide range of reports and charts that can be filtered in various ways such as by user, group, component and so on. You can also change labels, perspective and certain other report properties, as well as adding custom fields you've defined. However, you don't have the ability to create your own reports. Because the database is a third-party product, you can use other tools such as Crystal Reports to create custom reports, but you'll have to figure out the database schema first.

With care and planning you should be able to define a functional and flexible workflow for your product life cycle. Best of all, DevTrack will help you develop a more repeatable and dependable process model.

Third-Party Integration
DevTrack ships with client and server components. The server manages the central database and the client is used to access it. By default, DevTrack uses a Microsoft Access database as its engine, but DevTrack can also be configured to use Microsoft SQL Server, Sybase SQL Anywhere or Oracle. Because DevTrack uses ODBC, other databases, such as Informix, can be used. Being able to choose your existing database really simplifies administration.

I initially installed DevTrack on my Windows NT workstation using the default Access database. The installation was smooth and trouble-free. Then my NT box went wonko (this was not DevTrack's fault) and I had to move to my Windows 2000 machine. This time I decided to use Microsoft SQL Server. Four hours later, I finally had it working. The problem was that TechExcel provides almost no documentation (half a page in the manual and some help screens) explaining how to do so. Although it's not particularly difficult to do, important information like the name of the ODBC data source, and the name and password for accessing the database should be included in the documentation, rather than buried in a FAQ on the company's Web site.

DevTrack provides an interface to Visual Source Safe (VSS) that enables developers to associate issues with specific code modules and to check them in and out from DevTrack. This is handy. For instance, a New Feature is submitted and assigned to developer X. He implements it and links the New Feature issue to the appropriate module in VSS. Then he submits the feature for testing, at which point a bug is detected. The tester generates a Bug issue, links it to the original New Feature issue and submits it back to development. Because the original developer is on vacation, the project manager assigns the bug to someone else, who can then trace from the Bug issue to the New Feature, and hence to the code module. As I said, handy.

But developers tend to be IDE-centric, and so the add-on that allows developers to access DevTrack from Visual Studio is potentially even more handy. However, this tool is sold separately, and I didn't have the opportunity to test it. TechExcel is also working on integration with the Perforce SCM, and this capability should be available by the time you read this.

There are no means provided to directly integrate DevTrack with a testing tool. DevTrack can accept new issues via e-mail (a nice touch), so if your testing tool can generate customized e-mail messages, you can have it e-mail DevTrack. Or perhaps you could persuade your testing tool to access the DevTrack database directly (although this is potentially dangerous). But direct support for testing tools like the ones from Rational or NuMega would be highly useful.

Letting Outsiders In
DevTrack Web is an add-on product that provides almost all user capabilities via the Internet and a Web browser. The Web interface is excellent. (Indeed, I found all interfaces to the program were well-designed—at least to a programmer.)

I mentioned the e-mail support previously. At first I wasn't sure how useful this would be. But then, as I was thinking about integration with NuMega's tool suite, I remembered how useful I found Fail Safe's e-mail feature. Here's the idea: You add e-mail capability to the program you're developing, and whenever your error-trapping code detects a problem, it e-mails DevTrack with a problem issue. When I used Fail Safe's e-mail capability in a program a few years ago, my clients adored me because I was so responsive to problems during the testing phase. The format for reporting a bug via e-mail is straightforward, as is adding e-mail to most programs—and you are doing error-trapping, aren't you?

ClearQuest
Rational Software's ClearQuest 2001A also comes with paper manuals, as well as a documentation CD-ROM. ClearQuest's printed documentation consists of two short introductory booklets and two larger manuals that cover use and administration of the product. These documents are far more useful than those provided with DevTrack, but still omit critical information, such as the fact that when configuring ClearQuest to access a SQL Server database, you need to specify the same name for each of the requested user names.

Like DevTrack, ClearQuest supports Microsoft Access by default and can also use Microsoft SQL Server and Oracle, as well as IBM's DB2. Unlike DevTrack, ClearQuest doesn't use ODBC, and so using another database is impossible. Like DevTrack, configuring the product to use Access is easy, and configuring it to use SQL Server more difficult. Unlike DevTrack, I eventually had to call Rational to accomplish it. Again, once you know how, it's easy.

ClearQuest also supports a workflow model for development. The workflows are defined in what Rational calls a schema, and the company provides several of these with the package, ranging from simple defect-tracking to schemas that provide tight integration with Rational's other products. The provided schemas are completely customizable. If none of them satisfy your needs, you can design one from scratch, specifying states, state transitions, actions, behaviors, fields, record types, forms and just about anything else you can think of, including the addition of VB or Perl scripts. The model is tremendously powerful and flexible. But as is usually the case, power and flexibility come at a price.

Because the schema model is so complex, making even minor changes to an existing schema requires some thought. For instance, I added a field to the defect form to collect a description of the way the defect was fixed. I wanted to make this field mandatory when a defect was resolved—which was easy enough to do, but then I discovered there were state transitions that could bypass the resolution state, and it took some careful thought to produce the behavior I wanted. Rational really needs to include an extensive tutorial manual dedicated to schema design in the package, because creating a schema from scratch with the information provided is tremendously difficult. Nevertheless, I was impressed with the elegance of Rational's approach.

Playing With Others
As you probably know, Rational publishes a suite of development tools that cover all aspects of the development cycle. And, as you might suspect, ClearQuest can integrate very nicely with the suite. Rational provides what it calls packages. These offer specific integration with tools like Rational's ClearCase, RequisitePro, TeamTest and others. A package is a plug-in component that may be added to a schema, providing the records, forms, behaviors and so on that are needed for the integration. The packages can make customizing an existing schema much easier.

Via another package supplied with ClearQuest, the program can be integrated with Visual SourceSafe. I didn't explore this option because I didn't have VSS on the test machine (and, as I mentioned, my NT box had gone south). Nevertheless, the support for VB script means that any application that publishes a COM interface can be integrated with ClearQuest. This is an excellent solution to the need to use third-party programs.

ClearQuest includes Web support via Microsoft's Internet Information Server 4 or 5 and a Web browser. As with DevTrack, most client capabilities are available via the Web, but I didn't find the interface as easy to use as DevTrack's. For that matter, I didn't find the client interface as easy to figure out, either. Like DevTrack, users can also submit issues and can be notified of ClearQuest events via e-mail.

ClearQuest offers an extensive array of predefined reports and charts, along with a query editor that enables you to create new reports and charts and modify existing ones. And, if that's not enough, Rational includes Crystal Reports 8.0 in the package, enabling you to issue queries directly against the database—figuring out the database structure is another matter entirely.

Either/Or
DevTrack and ClearQuest are complete and well-designed products. Both are suitable for tracking the entire life of a development project and providing some of the metrics needed to improve your development process. Both tools are highly configurable and can likely be adapted specifically to your needs. So which to choose?

DevTrack's two biggest advantages are its ease of use and its price—even with the Web add-on, it's much less pricey than ClearQuest. The ease of use applies to the client and Web UIs, and to customizing the product. I quickly got up-to-speed and became comfortable with the product.

However, if you're already using Rational products or thinking about investing in some, ClearQuest is the obvious choice. Additionally, ClearQuest is significantly more configurable than DevTrack and has better reporting capabilities. Furthermore, if you have existing tools that you'd like to integrate with your change- and defect-tracking system, and if they publish a COM interface, ClearQuest is a better choice.

On the whole, though, you won't go wrong with either tool.

DevTrack 4.1
TechExcel Inc. 3400 Mt. Diablo Blvd., Ste. 200 Lafayette, CA 94549 Tel: (925) 871-3900 Toll-free: (800) 439-7782 Online: www.techexcel.com Price: Web server, $1,995; single user, $595; 2-10 users, $545 per user. Technical Requirements: Windows 2000, NT and 95/98
RATING: ***	The Rate Sheet
Pros: DevTrack is highly configurable. It's easy to use. It's reasonably priced.	Cons: It provides a limited number of reports. There is little provision for interacting with other tools. Documentation is poor.